Challenge
Construction is one of the most dangerous industries to work in, and skilled machine operators are in short supply. Caterpillar has worked with customers on autonomous mining operations for more than 15 years. Now it’s bringing that experience to construction sites.
Only a few years ago, Caterpillar didn’t know when autonomy in these environments would be technically possible. Models that can perceive their surroundings, reason, and act are bringing it within reach. Training them starts with real-world machine data that no model understands on its own, and a lot of it.
To move forward, Caterpillar needed to:
- Synchronize terabytes of camera, lidar, control, and operator data from a single machine every day
- Cut the weeks of manual annotation and labeling before data was ready to use
- Secure the GPU capacity for a large training job, along with expertise it was still building in-house
“You take the combination of [NVIDIA’s] platform plus the expertise that CoreWeave has in being able to scale, operate, and apply that. That’s where this collaboration between NVIDIA, Caterpillar, and CoreWeave really starts to come together to help us do some of these things that really haven’t been done in the industry before.”
Brandon Hootman, VP of Physical AI Platforms and Construction Autonomy
Solution
Caterpillar began working with CoreWeave on physical AI and construction autonomy in 2026. It chose CoreWeave for two reasons: the GPU capacity to train on huge volumes of machine data, and a team that knows how to put physical AI to work.
Automated data annotation and labeling
Working with CoreWeave and NVIDIA, Caterpillar uses AI models to annotate and label incoming machine data, describing what’s happening around each machine.
Infrastructure for the physical AI loop
While training gets most of the attention, the majority of the work happens in the loop around it. Data comes from real machines and is stored and preprocessed at scale. Simulation runs on its own class of compute, reinforcement learning runs on top of simulation, and a better policy goes back to the mining machine. How fast Caterpillar moves through that loop determines how fast the machines improve.
Capacity for simulation and synthetic data
Field collection alone can’t cover every scenario a machine will face. For every hour of real data Caterpillar collects, it expects to generate roughly 1,000 hours of simulated data. Training, simulation, and synthetic data generation are its main workloads today.
Training at physical-world scale
Teaching an existing model what a job site looks like, or how to run a machine’s controls, is a large training job. Training is the largest share of Caterpillar’s physical AI work, with data preparation, management, and storage alongside every run. As capabilities reach machines in the field, those models will run on board, where machines can’t count on network connectivity.
Capacity plus people who know the technology
Caterpillar brings the machine and industry expertise, and CoreWeave brings the capacity and the people who know how to apply the technology. Brandon Hootman, VP of Physical AI Platforms and Construction Autonomy at Caterpillar, calls it “a really good marriage between the two.”
From Fully Connected: Caterpillar, NVIDIA, and CoreWeave discuss physical AI at industrial scale and what it takes to move from proof of concept to fleets in the field.
Outcomes
- The work is structured around what Caterpillar’s customers need: autonomy that does the work as efficiently as possible, including fuel efficiency, and fits the way they already run their job sites. While it’s early in the partnership, the foundation is in place to scale autonomy across Caterpillar’s fleet.
Experts’ time spent on valuable work
Caterpillar has moved labeling that once required teams of people reviewing recordings onto an AI pipeline built with CoreWeave and NVIDIA. Work that took weeks now takes hours, and Caterpillar’s specialists spend their time improving the models rather than tagging data
One portfolio, many machines to train
Caterpillar wants new AI-powered capabilities on as many of its machines as possible, with each machine type adding its own training work. CoreWeave gives it room to keep training new machines as the fleet grows.
Faster development, built with customers
Across the industry, development cycles that once took years are now measured in months. Caterpillar is building in a modular, iterative way so they can co-develop with customers to find where autonomy adds the most value. Functional safety systems, like the payload and grade systems already on the machines, stay in place as essential guardrails while new models are added.
